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Executive Summary 


The goal of this study is to determine what makes successful schools different from other 
schools. Rather than define success in absolute terms — such as the percentage of students who 
are proficient on a standardized test — this study’s definition is based on whether or not a school 
is performing better than predicted given the characteristics of the students it serves. Using data 
from over 1,700 California public middle and high schools, 40 schools were identified that 
consistently performed better than predicted on standardized tests of math and English language 
arts achievement. These schools were labeled “beating-the-odds” (BTO) schools. 

A previous study using this definition of success found that personnel resources — such as the 
education, experience, and roles of staff — did not help distinguish successful from unsuccessful 
schools. The current study looks at the relationship between school climate and success, as 
measured by the California Healthy Kids Survey. The measure includes such dimensions of the 
school environment as safety, academic supports, social relationships, and school connectedness. 
A positive school climate has been associated with higher academic achievement and healthy 
behavioral outcomes for students. 

The results of this study show that BTO schools had substantially more positive levels of school 
climate than other schools. BTO schools had climate scores at the 82 nd percentile, on average, 
whereas other schools were at the 49 th percentile, on average. Differences in school climate were 
twice as large between BTO schools and 20 schools that were consistently performing worse 
than expected (“chronically underperforming”). 

These differences were attenuated, but remained large and significant, even after adjusting for 
schools’ student characteristics and personnel resources. In other words, school climate 
distinguished BTO schools from non-BTO schools that served the same types of students and 
had the same types of staff. Furthermore, school climate was more strongly associated with the 
likelihood of beating the odds than student demographics or personnel resources. 

This study adds to the growing body of evidence suggesting that school climate is an important 
factor for school success. The study builds on the school effectiveness and school climate 
literature by examining not only if there is an association between climate and achievement — as 
previous research has done — but by examining if school climate helps understand how a subset 
of schools is consistently able to beat the odds and perform better than its peers. 

School climate is a malleable factor that schools or districts are able to manipulate. Expansion of 
programs designed to improve school climate could result in increased success for a broader 
number of schools. School climate may be part of the solution to helping schools beat the odds. 
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Chapter 1 . Introduction and Overview 


Introduction 

Amid a national education narrative fraught with themes of dropout factories and failing schools, 
the search for successful public schools and turnaround strategies has been in earnest. Education 
scholars and policymakers invoke the term “pockets of excellence” in discussing public schools 
that stand out above their peers. These schools are seen as ideals for public schooling, and other 
schools can learn from their successes. 

In 2007, the American Institutes for Research (Perez et al., 2007) published Successful 
California Schools in the Context of Educational Adequacy (“2007 SCS report,” hereafter). The 
report compared academically successful and unsuccessful California schools — based on 
whether they did better or worse than would be predicted on standardized tests — and sought to 
understand how personnel resources might explain the difference between the two. The 2007 
SCS report concluded that, in regard to using personnel resources to distinguish successful from 
unsuccessful schools, “traditional resource measures do not seem to be what are making the 
difference” (p. 89). 

What, then, is making the difference for successful schools? This report, A Climate for Academic 
Success, approaches this question by using a “successful-schools” methodology similar to that of 
the 2007 SCS report, but it examines an alternative set of explanatory factors. This study 
explores differences in school climate as well as personnel resources. The specific questions 
addressed in this report are: 

■ What are the reported differences in school climate by students in academically 
successful versus unsuccessful schools? 

■ What are the relative associations between the likelihood of school success, on the one 
hand, and school climate and personnel resources, on the other? 

■ What are the practical implications of these findings for improving the academic 
performance of schools? 

Successful Schools 

What defines a successful school? A school’s performance depends heavily on its student body. 
The Coleman Report (Coleman et al., 1966) famously drew the connection between students’ 
socioeconomic status (SES) and academic achievement. This relationship, as well as the 
relationships between other student characteristics (e.g., race, English learner status) and 
achievement, has been reaffirmed many times since (see Duncan & Murnane, 2011 for a 
collection of recent work on inequality in education). For this reason, it is problematic to use an 
absolute performance measure to diagnose success. Schools that perform well typically enroll 
students with higher incomes and fewer special needs. 

To illustrate this point, consider California’s measure of school success — the Academic 
Performance Index (API). According to the state accountability system, a school must attain an 
API of 800 or higher to be deemed successful. Exhibit 1.1 shows the 1,331 public middle and 
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high schools in California that meet this criterion and the 5,349 that do not. It also shows the 
percentage of students in those schools who are eligible for the free and reduced-price meals 
program (FRPM), a proxy for poverty, and those who are English learners. 

Exhibit 1.1. Demographics of California public middle and high schools, by API 



School level 

N 

% Free and 
Reduced- 
Price Meals 

% English 
Learners 

School at or 

Middle 

837 

24.4 

8.2 

above 800 API 

High 

Total 

494 

1,331 

18.7 

22.3 

5.2 

7.1 

Schools below 

Middle 

1,865 

66.4 

25.2 

ann api 

High 

3,484 

52.0 

18.0 

OUU Mil 

Total 

5,349 

57.0 

20.5 


Note: Statistics were derived from the study sample described in Chapter 3 of this report. 

Schools that meet the API threshold for success have less than half the proportion of students in 
poverty, on average, compared to schools with APIs below 800 (22.3 percent versus 57.0 
percent). Furthermore, schools with APIs below 800 have three times as many English learners, 
on average. 

One must keep the relationship between student demographics and achievement in mind when 
linking any other school characteristics with success. For example, there is a demonstrable 
connection between a school’s climate and its API (O’Malley & Hanson, 2012), but it is likely 
that the SES of the school’s students underlies both of these factors (see Gottfredson, 
Gottfredson, Payne, & Gottfredson, 2005, for evidence of the connection between student SES 
and school climate). It is plausible that high SES students — and their wider access to resources — 
are the true engine of both higher achievement and better climate in schools. Failure to account 
for SES confounds the relationship between achievement and school characteristics like climate. 

To account for this potential confound, the study employed the definition of success used in the 
2007 SCS report that takes student demographics into account. Rather than absolute performance 
on a measure like the API, success is defined by a school’s performance relative to how it would 
be predicted to perform based on its student population. Comparing a school’s actual and 
predicted performance indicates which schools are performing better than predicted and which 
are performing worse than predicted. The successful schools described in this report are not 
those that have the highest test scores in the state; rather, they are “ beating the odds ” (BTO) 
schools that outperform other schools with similar types of students. The unsuccessful schools 
are labeled “chronically underperforming” (CU). A detailed description of the methodology used 
to develop this criterion is provided in Chapter 3. 

School Climate 

School climate has become increasingly popular in discussions of school success. School climate 
is a broad term used to describe the school environment, and while it has no consensus 
definition, reviews of the topic have identified several recurring themes: (a) order, safety, and 
discipline; (b) academic supports; (c) personal and social relationships; (d) school facilities; and 
(e) school connectedness (Austin, O’Malley, & Izu, 2011; Cohen, McCabe, Michelli, & Pickeral, 
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2009; Zullig, Koopman, Patton, & Ubbes, 2010). There is evidence to suggest that these factors 
have an important role in turning around unsuccessful schools (Bryk, Sebring, Allensworth, 
Luppescu, & Easton, 2010), and a positive school climate has been associated with higher 
academic achievement and healthy behavioral outcomes for students (Brand, Felner, Shim, 
Seitsinger, & Dumas, 2003; Patton et al., 2006). 

This study benefits from the availability of widespread data on school climate in California. 
Between 2003 and 2011, California districts that received funding through the Safe and Drug- 
Free Schools and Communities Act, Title IV, Part A of the No Child Left Behind Act of 2001 or 
the state Tobacco Use Prevention Education program were required to administer the California 
Healthy Kids Survey (CHKS), which is largely focused on measuring school climate. 
Approximately two thirds of all public middle and high schools in the state administered the 
CHKS. These data are used to create a global school climate score for each school in which the 
CHKS was administered, as well as subscale scores that measure specific dimensions of school 
climate. 

Organization of the Report 

Chapter 2 of this report examines the existing research literature on school climate and its 
relationship with school success. A body of evidence links positive school climate with improved 
student academic outcomes; however, no studies use a successful-schools approach. 

Chapter 3 describes the data used in the study, the method of identifying successful and 
unsuccessful schools, and the method for analyzing the relationships between school climate and 
school success. 

Chapter 4 presents the results of the analyses, demonstrating a strong and significant positive 
association between school climate and school success. The analyses also suggest that school 
climate has a stronger relative association with school success than a school’s personnel 
resources. 

Chapter 5 includes concluding remarks on the study and implications for future research and 
practice in light of the findings. 
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Chapter 2. What We Know About School Climate 
and School Success 


Introduction 

This chapter contains three sections that review areas of previous research relevant to the present 
study. 

■ Section 2-A provides a brief overview of the successful-schools approach. 

■ Section 2-B examines literature on school effectiveness to identify a set of school 
characteristics associated with success. 

■ Section 2-C summarizes studies that have explored the connection between school 
climate, student achievement, and school success. These reviews lay the foundation for 
the analysis of successful schools in the context of school climate. 

Section 2-A. Successful-Schools Approach 

The successful-schools approach has traditionally been used by education finance scholars to 
determine appropriate expenditure levels. Once success is defined, the next step is to determine 
how much successful schools are spending, with this spending figure becoming a sort of norm or 
expectation for the resources required to make schools successful. The 2007 SCS report 
summarizes and critiques a body of research literature that employs the approach, reaching two 
relevant conclusions. 

First, previous studies using the successful-schools approach define success in absolute terms 
(e.g., a certain percentage of students above a certain proficiency level) or in terms of growth 
rates (e.g., “adequate yearly progress”). Whatever the successful schools are spending is then 
considered “adequate” funding for all schools. This approach’s simplicity has appeal for 
researchers and policymakers alike: “Here’s what successful schools are spending,” the logic 
goes. “Let’s make sure all schools have that level of resources.” 

Second, the 2007 SCS report concludes that using absolute criteria to define success is overly 
simplistic and ignores the impact of student characteristics on schools’ resource needs. When 
using an absolute selection criterion, the successful schools will typically be those with largely 
affluent student populations. Schools that serve students who are predominantly low income and 
special needs likely need more resources on a per-student basis. If these types of schools are 
systematically excluded from being identified as successful, the results of any analysis linking 
resources and success will be biased. 

The present study takes advantage of the elegance of the successful-schools approach, while 
avoiding the pitfalls pointed out in the 2007 SCS report. It does this by defining success in terms 
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that take schools’ student populations into account. It also adds a new twist to research in the 
successful-schools mold, treating school climate, rather than resources, as the concept of focus. 

Section 2-B. Selected Literature on Elements of School 
Effectiveness 

While the successful-schools approach represents a specific method for determining what makes 
schools effective, a rich body of research in this area uses alternative approaches. In large part, 
the attempt to identify elements of effective schools grew from the findings of the Coleman 
Report (1966) that stressed student demographics as the most salient predictor of academic 
success. The social forces that render students of particular backgrounds less successful in school 
are difficult for educators to address. Therefore, researchers focused on what schools could do in 
spite of those forces. 

Purkey and Smith (1983), in an oft-cited review of early school-effectiveness research, identified 
“school culture” as the most important factor in a school’s ability to facilitate student 
achievement. According to the authors, school culture is both a structure and a process. 

Structural factors that promote positive school culture include: 

• Administrative leadership 

• District support 

• Staff stability and development 

• Parental involvement 

• A strong curriculum with maximized learning time 

• Recognition of academic success 
Process factors include: 

• Staff collegiality 

• Sense of community 

• High expectations for students 

• Order and discipline 

Recent research on school effectiveness has yielded similar results. In a review of over 300 
studies commissioned by the U.S. Department of Education (Vishner, Emanuel, & Teitelbaum, 
1999), the predictors of effectiveness were high expectations, parent involvement, staff 
development, and more intimate learning environments, along with initiatives to connect 
learning with students’ interests and career plans. Factors identified by Marzano (2003), in a later 
review, overlapped heavily with those already mentioned. 

These findings have been further corroborated by a comprehensive empirical study of Chicago 
Public Schools by Bryk and colleagues (2010). In it, they used longitudinal data from a large 
sample of Chicago schools during a period of decentralization that created a laboratory for 
reform initiatives. A framework of essential supports for school improvement emerged from this 
work that included: 
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• School leadership 

• Parent-community ties 

• Professional capacity 

• Student-centered learning climate 

• Instructional guidance 

These five supports were interrelated, and the presence of all five greatly increased the likelihood 
of school success. 

In the 2007 SCS report, interviews were conducted with school principals at successful schools 
to determine, in an inductive manner, what they were doing differently compared to less 
successful schools. The results suggested three mediating factors: 

• High-quality staff 

• A standards-based curriculum 

• Coherent instruction 

These factors were, in turn, influenced by seven other factors: 

• Staff development 

• Principal control over hiring 

• Staff collegiality and cooperation 

• Assessment data for student achievement 

• Student services 

• Parent involvement 

• High expectations 

The commonalities across these reviews and studies regarding the qualities of effective schools 
are abundant. For decades, research on effective schools has boiled down to a handful of factors: 
leadership, relationships, and teaching and learning, to put it most generally. Purkey and Smith 
(1983), some 30 years ago, referred to these factors collectively as school culture or school 
climate, and these terms have regained prominence in education only in the last several years. 
Alternatively, some of the identified factors relate to personnel resources, such as teacher and 
administrator experience and the presence and number of student services professionals in a 
building. Personnel resources were the factor of interest in the 2007 SCS report. The focus here 
turns to school climate, defined in a way more consistent with Purkey and Smith’s process 
factors noted earlier. 

Section 2-C. Selected Literature on School Climate and 
Student Achievement 

School climate has gained in popularity for various reasons, not the least of which include its 
ability to organize disparate characteristics of schools under one theoretical umbrella and its 
amenability to intervention. The term has become something of a catchall for targets of school 
improvement, though several reviews have distilled its dimensions into a manageable framework 
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that includes (a) order, safety, and discipline; (b) academic outcomes; (c) personal and social 
relationships; (d) school facilities; and (e) school connectedness (Austin et al., 2011, Cohen et 
al., 2009; Zullig et al., 2010). 

A positive school climate has been associated with higher academic achievement and healthy 
behavioral outcomes for students. Brand and colleagues (2003) found multiple dimensions of 
school climate to be significantly associated with student academic, behavioral, and 
socioemotional well-being in a large, nationally representative sample of middle school students. 
The association between climate and achievement has been confirmed in data from California 
public middle and high schools (Hanson, Austin, & Zheng, 2011). Patton and colleagues (2006) 
conducted one of the few documented randomized trials involving school climate, showing that 
an intervention designed to enhance school connectedness and student engagement had the effect 
of lowering student health-risk behaviors and substance use. 

Due, in part, to its perceived malleability and, in part, to the alternative it presents to 
accountability-type reforms, school climate has been the focus of several recent federal 
initiatives for school reform, including the Safe and Supportive Schools program (Learning First 
Alliance, 2001). 1 

Contributions of the Present Study 

The present study builds on the school effectiveness and school climate literature by bringing the 
successful-schools approach to bear on the association between climate and performance. This 
study generated a definition of success that takes student demographics into account, per the 
recommendations of the 2007 SCS report. 

The study then examines differences in school climate between successful and unsuccessful 
schools. This represents a novel approach to school-climate research that is not only able to 
detect an association between climate and achievement — as previous research has done — but to 
examine whether school climate helps to explain how a subset of schools is consistently able to 
beat the odds. 

A final original contribution of this study is its comparison of the relative associations of school 
climate versus personnel resources with the likelihood of school success. Controlling for 
personnel resources makes the estimated effects of climate on performance more robust, and 
comparing the relative effects of climate and resources offers a point of reference that is often 
lacking in school-climate research. 


1 California is one of eleven states that received federal Safe and Supportive Schools grants to improve the learning 
conditions and other school climate factors in low-performing high schools. For more information and to access a 
wide range of resources to assist in school climate improvement, visit the project website: 
www.californiaS3.wested.org. 
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Chapter 3. Data and Methodologies 


Introduction 

This chapter describes the data sources and research methodologies used to conduct the study. 

■ Section 3-A addresses the process of identifying BTO and CU middle and high schools 
in California. 

■ Section 3-B describes the process of building a statewide, school-level database of school 
climate, personnel resources, and demographics used in the study analyses. 

■ Section 3-C describes the multivariate regression approach to assessing differences in 
school climate between BTO, CU, and other schools. 

■ Section 3-D describes the logistic regression approach to assessing the relative 
association with BTO status of school climate, personnel resources, and other school 
characteristics. 

Section 3-A. Identification of Beating-the-Odds and 
Chronically Underperforming Schools 

Data Sources 

To define school success, this study drew on data from the sources described below, which are 
available for all public middle and high schools in California. The sample was limited to schools 
that administered the CHKS (the measure of school climate) during the 2008-09 and 2009-10 
school years. During this period, the CHKS was administered in a representative sample of 1,715 
public middle and high schools, roughly two thirds of the state total. Only traditional middle and 
high schools were included in the study; the single charter school that met the above criteria was 
excluded from the analyses. 

Student and Demographic Characteristics 

Several databases were combined to create a profile of public middle and high schools in 
California from 2007-08 to 2010-11. Information regarding schools’ racial composition, 
proportion of English learners and students with disabilities, and students receiving FRPM was 
drawn from the California Basic Educational Data System (CBEDS) API database. 

Outcome Measures 

Academic performance indicators were accessed through the California Standardized Testing 
and Reporting (STAR) program. The academic outcomes of interest were middle school English 
language arts (ELA) and math scores on the California Standards Tests (CST) from 2007-08 to 
2010-11 and high school ELA CST and math scores in grade 10 on the California High School 
Exit Examination (CAHSEE) from 2008-09 to 2010-11. CAHSEE math scores were used in lieu 
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of CST math scores in high school because CST scores are course- specific (e.g., algebra, 
geometry) and thus are not readily comparable. The CST and CAHSEE are major components of 
a school’s API. For each school, an overall student average for each of these test scores was 
calculated, as well as an average for each of the following subgroups: African American and 
Hispanic students, English learners, and students receiving FRPM. 

Methodology for Selection 

Our approach to classifying schools as BTO or CU parallels that of the 2007 SCS report. Due to 
instability in schools’ average test scores from year to year, BTO/CU classification was based on 
multiple years of data. Specifically, four years of student demographic data and test scores for 
middle schools and three years for high schools were used. CST and CAHSEE scores were 
standardized by grade level to allow for cross-grade-level comparability and were then weighted 
by the relative number of students tested in each grade to create a school-level average academic 
performance measure. This process was repeated for each of the aforementioned subgroups, as 
well. 

BTO and CU schools were identified based on the difference between their actual academic 
performance and their predicted performance in consideration of their demographic makeup. A 
series of linear regression models were estimated to determine this predicted academic 
performance score: 


Y tP iy = P tV Q + + £ tP iy 


where Y is the academic performance outcome variable in subject t, for student group/subgroup 
p, in school i in year y. A is a vector of school-level student demographic characteristics that 
includes the proportion of students who (a) receive FRPM, (b) are English learners, (c) have 
disabilities, (d) are Hispanic, (e) are African American, and (f) are Asian. Each school’s fitted 
value ( Y tp iy ), predicted by these regression models, was subtracted from its actual academic 
performance, resulting in a difference score, or a residual, s tp iy . This residual reflects how well a 
school is performing versus how well it is predicted to perform based on its student body. A 
negative residual indicates that a school is performing worse than predicted for that subject, 
student group/subgroup, and year; a positive residual indicates that a school is performing better 
than expected. 

Beating-the-Odds Schools 

To be classified as BTO, a school had to meet several criteria, summarized in Exhibit 3.A.I. 

First, a school’s residual had to be 0.25 standard-deviation units 2 above the mean residual to be 
considered “overperforming” (for each subject, student group, and year). For schools that 
performed roughly as predicted (i.e., have a residual near zero), there was little difference in 
performance between those that performed worse than predicted versus better than predicted, and 
a 0.25 standard-deviation criterion better distinguished the overperforming versus 


2 In the 2007 SCS report, the cut-point used as part of the BTO-selection criteria was 0.75 standard-deviation units 
above the mean. Due to the smaller sample of schools in the present study, this standard on inclusion resulted in 
only 13 schools being identified as BTO. Thus, this criterion was relaxed in order to classify a greater number of 
schools as BTO. 
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underperforming schools. For example, one school’s residual could be just below zero and 
another school’s residual could be just above zero. While these two schools would be performing 
nearly identically, without this criterion the former’s performance would be considered 
underperforming and the latter’s overperforming. 

Exhibit 3.A.I. Criteria for selecting beating-the-odds schools 


Middle schools 

For all students: 

■ CST ELA and math residuals for all students are more than 0.25 standard deviations above the mean 
residual for students in all middle schools in California for four consecutive years. 

For subgroups: FRPM, English learner, Hispanic, and African American students: 

■ CST ELA and math residuals for these subgroups of students are more than 0.25 standard deviations 
above the mean residual for each subgroup of students for four consecutive years. 

■ CST ELA and math residuals can be missing. The school may not have that subpopulation of students 
(or an insufficient number of students to be considered accountable for that subgroup) and still may be 
part of the pool of schools that are beating the odds. 

High schools 

For all students: 

■ CST ELA residual and CAFISEE math residual are more than 0.25 standard deviations above the mean 
residual for students in all high schools in California for three consecutive years. 

For subgroups: FRPM, English learner, Hispanic, and African American students: 

■ CST ELA residuals and CAFISEE math residuals for each subgroup are more than 0.25 standard 
deviations above the mean residual for each subgroup for three consecutive years. 

■ CST ELA residuals and CAFISEE math residuals can be missing. The school may not have that 
subpopulation of students (or an insufficient number of students to be considered accountable for that 
subgroup) and still may be part of the pool of schools that are beating the odds. 


Second, schools had to be considered overperforming in both ELA and math, overall and across 
four subgroups: FRPM, English learner, Hispanic, and African American. The subgroup 
requirement was designed to ensure that a school’s residual was not inflated or deflated by 
certain subgroups of students in the school. If a single subgroup was not overperforming, the 
school was not considered BTO. If a school did not serve a sufficient number of a certain 
subgroup of students to be considered accountable for that subgroup, then this subgroup’s 
performance was not factored into BTO selection. 

Third, as mentioned above, because test scores are relatively unstable from year to year, a school 
must have been overperforming — for each subject and student group — for three or four 
consecutive years (for high and middle schools, respectively). 3 A school could witness a one- 
year aberration in test scores (for all its students or for a subgroup of students) that is not 
reflective of its usual performance. To reduce error associated with this random variability. 
Exhibit 3.A.2 shows the number of public middle and high schools that met these criteria each 
year. Out of the total 1,715 schools, this selection process resulted in 40 schools (2.3 percent) 


3 BTO selection was based on four years of data — 2007-08 through 2010-11. School climate data from the CHKS 
were collected in either 2008-09 or 2009-10, depending on when the school administered the survey. Thus, only two 
of the four years of achievement data correspond directly with CHKS data. The goal of using additional years of 
achievement data was simply to make the BTO selection process more robust, and this decision does not reflect any 
assumptions about causality between BTO status and school climate. 
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being classified as BTO, 17 of which were middle schools and 23 were high schools. 4 The 40 
BTO schools consistently performed much better than predicted for all subgroups of students. 

Exhibit 3.A.2. Number of overperforming schools each year, 2007-08 to 2010-1 1 



2007-08 

2008-09 

2009-10 

2010-11 

All years 

Middle schools 

88 

104 

106 

112 

17 

High schools 

N/A 

78 

81 

70 

23 

Total 

88 

182 

187 

182 

40 


Exhibit 3.A.2 also illustrates the instability of test score results from year to year. A school that 
was overperforming one year was unlikely to sustain that performance for three or four 
consecutive years. For example, only 17 of the 106 middle schools (16 percent) that were 
overperforming during the 2009-10 school year met the four- year criteria for BTO. 

Chronically Underperforming Schools 

The selection process for CU schools mirrored that used for identifying BTO schools, except that 
the residual cut-point was at least 0.25 standard-deviation units below the mean for each subject, 
student group, and year. Twenty schools (1.2 percent) were identified as CU using these criteria: 
13 middle schools and 7 high schools. These 20 schools consistently performed worse than 
predicted for all students. 

Selected Schools 

In any given year, BTO schools performed better than predicted on tests of achievement (i.e., 
they had high residual scores), and CU schools performed worse than predicted on tests of 
achievement (i.e., they had low residual scores). Exhibit 3.A.3 shows this distribution of 
residuals for all 1,715 schools, using 2009-10 CST ELA scores as an example. BTO schools are 
represented by black diamonds, CU schools by white diamonds, and other schools by light gray 
diamonds. The horizontal axis indicates a school’s API in 2009-10. The API is a single number, 
ranging from 200 to 1000, that reflects a school’s performance level based on the results of 
statewide testing. As noted in Chapter 1, a school must attain an API of 800 or higher to be 
deemed successful according to the state accountability system. This threshold is indicated with a 
dotted line. 

The exhibit shows that the majority of BTO and CU schools were not outliers in terms of their 
2009-10 CST ELA performance. Some schools that were not considered BTO have higher 
residuals than BTO schools due to the single-year presentation of CST ELA residuals, but these 
schools did not meet the multiyear criteria for BTO classification. Furthermore, this exhibit 
shows that in 2009-10 most BTO schools had APIs above 800 and no CU schools had APIs 
above 800. This distribution is somewhat expected, as a school must be relatively high- 
performing for three or four consecutive years to be considered BTO. If a school has a very low 
API, it is unlikely to be BTO, even if its predicted performance — based on student 
demographics — is low. However, it is notable that nearly half of BTO schools had APIs below 
800, the state threshold for success, in 2009-10. This situation reiterates that it is possible for a 


4 The study sample was limited to schools that had administered the CHKS in 2008-09 or 2009-10 (representing 
about two thirds of all California public middle and high schools). Among all public middle and high schools in the 
state, there were 108 that met the BTO criteria. 
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school to perform much better than predicted but still not perform well according to the absolute 
state standard. 

Exhibit 3.A.3. Distribution of ELA residuals across API for BTO, CU, and other schools 



♦ ♦ 

o 


500 600 700 800 900 1000 

API in 2009/10 


♦ BTO schools O CU schools 
Other schools 


The names of the 40 BTO schools are provided in Appendix A. Eight of the 40 (20 percent) are 
classified as magnet schools 5 by the National Center for Educational Statistics (Keaton, 2012), 
whereas only five percent of other sample schools are magnet schools. For some of the following 
analyses, parallel analyses were conducted with magnet schools excluded. There were no 
substantive differences in results when magnet schools were left out. 

To this point, a group of schools has been identified that consistently outperformed (i.e., BTO 
schools) and underperformed (i.e., CU schools) predictions. This result leads naturally to the 
question, “What distinguishes schools that beat the odds from those that do not?” The analyses 
that follow provide insight into this question. 

Section 3-B. The School Climate Database 

The factor of interest in explaining why some schools outperformed and underperformed 
predictions is school climate. Assessing the relationship between BTO and CU status and school 


5 The California Department of Education defines a magnet school as a public school that focuses on a special area 
of study that is designed to attract students from across the district. The state does not provide special funding for 
these programs. More information is available at http://www.cde.ca.gov/sp/eo/mt . 
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Exhibit 3.B.I. Data sources and variables used in analyses 


Data Source/Variable Name 

# 

Variable Description 

CBEDS - 2008-09 & 2009-10 

Academic Performance Index (API) 

FRPM 

1 

Percentage of students who are eligible for the free and reduced-price meals 
program 

African American 

2 

Percent Black or African American students 

Hispanic 

3 

Percent Hispanic or Latino/a students 

EL 

4 

Percent English learners 

Enrollment 

5 

Number of students enrolled on the first day of testing 

Professional Assignment Information Form (PAIF) 

Teacher education 

6 

Teachers’ highest education level 

Teacher experience 

7 

Teachers’ total years of public and/or private educational service 

Teacher tenure 

8 

Indicates that a teacher has tenure 

Teacher credentials 

9 

Indicates that a teacher is fully credentialed 

Administrator experience 

10 

Administrators’ total years of experience in education 

Student support staff 

11 

Indicates an employee of the district in a position requiring a standard 
designated services credential, a health or development credential, or a 
librarian credential and who performs direct services to pupils (e.g., 
counselors, guidance and welfare personnel, librarians, psychologists) 

CHKS - 2008-09 & 2009-10 

School Climate Index 

12 

Composite score of “supports and engagement” and “low violence, 
victimization, and substance use” domain scores 

Supports and engagement 
domain score 

13 

Composite score of four “supports and engagement” subdomains below 

High expectations and 
caring relationships 

14 

Construct of six survey items that measure students’ perceptions of the degree 
to which school staff have high expectations for student performance and have 
caring relationships with students 

Meaningful participation 

15 

Construct of three survey items that measure students’ perceptions of the 
degree to which their school provides opportunities for them to participate in 
activities and decision-making 

Perceived school safety 

16 

Construct of two survey items that measure the degree to which students 
report feeling safe at their school 

School connectedness 

17 

Construct of four survey items that measure the degree to which students 
report feeling connected and respected at their school 

Low violence, victimization, 
and substance use domain 
score 

18 

Composite score of four “low violence, victimization, and substance use” 
subdomains below 

Low violence perpetration 

19 

Construct of seven survey items that measure the frequency of student 
perpetration of violence, vandalism, and other crimes on school property 

Low violence victimization 

20 

Construct of six survey items that measure the frequency of student 
victimization of physical and emotional violence on school property 

Low harassment and 
bullying 

21 

Construct of five survey items that measure the frequency of student 
victimization of specific difference-based bullying and harassment on school 
property 

Low substance use at 
school 

22 

Construct of four survey items that measure the frequency of students’ use of 
various substances on school property 

Low truancy incidents 

23 

Percentage of students who missed more than 30 minutes of instruction 
without an excuse at least three times during the school year 
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climate required the construction of a database that included binary variables representing the 
BTO and CU classifications described in the previous section, school climate indicators derived 
from the CHKS, personnel resources data from the CBEDS Professional Assignment 
Information Form (PAIF), and school demographic data from the CBEDS API database. 

California Healthy Kids Survey 

A single administration of the CHKS was required of California public schools during the 2008- 
09 to 2009-10 period as a condition of Safe and Drug-Free School and Communities (Title IV) 
funding or the state Tobacco Use Prevention Education (TUPE) program. Approximately two- 
thirds of all traditional public middle and high schools in the state had students complete the 
survey. In one large district, only a small sample of the entire population of schools completed 
the survey. Other schools did not administer the survey due to not receiving Title IV or TUPE 
funding, being exempt from this requirement under the Rural Education Achievement Program, 
or for unknown reasons. 

The CHKS core module is a 1 15-item student survey that is primarily designed to measure 
school climate. In completing the CHKS, students indicate how safe, supportive, and engaging 
they perceive their school to be; how often they perpetrate or are victims of violence; and how 
often they use substances. A factor analytic study of the CHKS identified eight latent constructs, 
or subdomains (Hanson, 2012): 

• High expectations and caring relationships 

• Meaningful participation 

• Perceived school safety 

• School connectedness 

• Low violence perpetration 

• Low violence victimization 

• Low harassment and bullying 

• Low substance use at school 

These constructs load onto two more general constructs, or domains: (a) supports and 
engagement (constructs 1-4); and (b) low violence, victimization, and substance use (constructs 
5-8). These two domains (90 percent) combine with a reverse-coded truancy indicator from 
schools’ administrative records (10 percent) to calculate an overall school climate rating, or 
School Climate Index (SCI), which is rescaled to range from 100 to 500, with a mean of 300 and 
a standard deviation of 50. All of the school climate domains (e.g., support and engagement) and 
subdomains (e.g., high expectations and caring relationships) are scaled in a similar fashion. In 
previous research, a school’s SCI was found to be associated with its API (O’Malley & Hanson, 
2012 ). 
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California Basic Educational Data System 


Some schools administered the CHKS during the 2008-09 school year and some during the 
2009-10 school year, and school demographic data from the CBEDS API dataset for the 
corresponding year were linked to each school. This linkage resulted in a combined data file 
wherein all CHKS schools had same-year information regarding the percentages of students who 
were (1) receiving FRPM; (2) African American; (3) Hispanic; and (4) English learners; as well 
as (5) the number of students enrolled. 

To operationalize personnel resources — a factor found to be significantly associated with BTO 
status in the 2007 SCS report — staff characteristics from the PAIF were aggregated to the school 
level and merged with the CHKS/demographic database, again on a same-year basis (i.e., either 
2008-09 or 2009-10). Personnel resources included, for example, staff experience and education 
and the proportion of staff in student support services roles. The full set of personnel-resources 
variables is presented in Exhibit 4.A.2 in the following chapter. Only those personnel-resources 
variables found to be significantly associated with BTO or CU status in the 2007 SCS report (at 
the p < .05 level) were retained for analyses, along with administrator experience, for which 
there was a large difference in averages between BTO schools and other schools in the present 
data. 

As a next step, personnel-resources variables that were determined to be highly correlated (r > 
0.60) were removed to avoid problems of multicollinearity in subsequent regression models. In 
the case that two variables were identified as being collinear, the variable with the stronger 
correlation with the SCI was retained. For example, the school-level variables for the average 
years of teacher experience in education and the average years of teacher experience in the 
district were highly correlated (r = 0.90) and thus likely to lead to unreliable estimates if both 
were included as model variables. However, since the former variable had a stronger correlation 
with the SCI (r = 0.23 versus r = 0.20), it was retained, and the latter variable was dropped. The 
retained school-level personnel-resources variables were (1) average teacher education, (2) 
average teacher experience, (3) proportion of teachers with tenure, (4) proportion of teachers 
with full credentials, (5) average administrator experience, and (6) the proportion of overall staff 
who are in student support services roles. 

This merged dataset was combined with the BTO and CU indicators described in the previous 
section, resulting in a comprehensive database of school climate, performance, demographics, 
and personnel resources. Descriptive statistics for these variables and the omitted personnel- 
resources variables are provided in the next chapter. 

Section 3-C. Methodology for Assessing School Climate 
Differences in BTO and CU Schools 

This section describes the methodology used to assess the differences in school climate between 
BTO schools, CU schools, and other California public middle and high schools. The 
comprehensive dataset described in the preceding section was used for these analyses. 
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Twelve multivariate regression models were estimated, one for each of 12 school climate 
outcomes, using the general equation: 

CLIMATE s = ft + PiBTOs + ftCft + P 3 STU s + ft5Tft + £ s 

where the outcome CLIMATE for school s represents a different outcome in each of the 12 
models: the SCI (in Model 1); the domain score for support and engagement (in Model 2) or 
violence, victimization, and substance use (in Model 3); one of the eight subdomain scores (in 
Models 4 through 11, respectively); or the low truancy index (in Model 12). BTO s and CU S are 
binary variables indicating whether school s fits the respective performance criteria (0 = no, 1 = 
yes). STU s is a vector that includes the five student-demographic variables, and STF s is a vector for 
the six personnel-resources variables. ft is a constant intercept term, and ft, ft, ft, and ft are 
parameters to be estimated. e s is a stochastic error term representing the variance in the outcome 
that is not explained by the predictor variables. 

The predictor variables of interest to this analysis are the BTO s and CU S binary variables. The 
coefficients for these two variables, ft, and ft, detect whether there are significant differences in 
school climate between BTO, CU, and other schools. 

The predictor variables representing student demographics and personnel resources acted as 
controls. The 2007 SCS report established a relationship between personnel resources and BTO 
and CU classification. Personnel resources may likewise be associated with school climate, as 
schools with better climate tend to have higher rates of teacher retention (Ingersoll, 2001). Not 
controlling for personnel resources in such an analysis could result in the detection of a spurious 
relationship between climate and BTO and CU classification. Further, while the BTO and CU 
selection process took student demographics into account — indeed the residuals used as part of 
the selection criteria were derived in part from achievement scores predicted by student 
demographics — as shown in Section 4- A, BTO and CU schools are disproportionately 
distributed across certain demographic spectrums. Student demographics are associated with 
BTO and CU statuses. There is also empirical evidence that students with different demographic 
backgrounds have different perceptions of school climate (Koth, Bradshaw, & Leaf, 2008). 

As with any cross-sectional survey research design, there are variables not included in this model 
that may be correlated with the outcome and predictors variables, such as parent involvement or 
intrinsic student motivation. However, the inclusion of both student demographics and school 
personnel resources as covariates represents a more conservative and unbiased estimate of the 
association between school climate and academic performance than most previous research on 
the topic. 

Section 3-D. Methodology for Assessing Associations 
Between School Characteristics and BTO Classification 

In addition to multivariate regression analyses to assess differences in school climate among 
BTO, CU, and other public middle and high schools, a logistic regression model was estimated 
that compares the relative associations of school climate and personnel resources, on the one 
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hand, and the likelihood of a school being classified as BTO, on the other. The comprehensive 
dataset described in the Section 3-B was used for this analysis. 

The equation used to estimate the logistic regression was: 

l Pr (BTO s ~) \ 

ln ( l - p r (BTOsV = P ° + PlSCIs + ^ + PzSTUs + PaSTFs + £s 

where the outcome is the natural log of the odds (“log odds”) of a school being classified as 
BTO. Log-odds units have little intuitive value unto themselves, but they allow for the estimation 
of linear relationships between continuous predictor variables and binary outcome variables, 
such as the BTO variable (Long & Freese, 2006). After estimating the logistic regression model, 
the probability of a school being classified as BTO can be calculated for specified values of the 
predictor variables. These probabilities are much more intuitive than log-odds units. 

Aside from the log-odds outcome, the logistic regression equation looks similar to the equation 
used to estimate the regression models in Section 3-C, except that the SCI has replaced BTO or 
CU status for school s as a predictor variable. The variables representing student-demographic 
and personnel-resources predictor variables remain the same, except that, instead of vectors, they 
represent a single index that is an average of a school’s standardized scores for all personnel- 
resources variables and student-demographic variables. In creating these two indices for 
personnel resources and student demographics, individual variables that were determined to be 
negatively associated with the likelihood of BTO classification were reverse-coded. Those 
variables were (a) the proportion of FRPM students and (b) enrollment, which both factored into 
the student-demographic index. 

For ease of comparison, all of the continuous variables included in the model (i.e., all predictor 
variables) were standardized so that their means equal 0 and their standard deviations equal 1 . 
This standardization allows the coefficients of all the predictor variables to be measured against 
one another to determine the relative strength of their association with the likelihood of being a 
BTO school. The next chapter presents the results of these analyses. 
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Chapter 4. School Climate Analysis Results 

Introduction 

This study’s objective is to determine whether there is a relationship between a school’s climate 
and its academic performance being better or worse than predicted based on its student body. 

The results suggest that this relationship exists. 

This chapter is divided into three main sections that mirror those of Chapter 3. 

■ Section 4-A describes how BTO, CU, and other schools differ in terms of demographics, 
personnel resources, and school climate (refer to Chapter 3 for an explanation of how 
BTO and CU schools were selected). 

■ Section 4-B presents the results of a series of multivariate regression models that tested 
for differences in school climate between BTO, CU, and other schools and take into 
account schools’ student demographics and personnel resources. 

■ Section 4-C presents the findings of the logistic regression model that juxtaposes the 
associations between school climate and personnel resources, on the one hand, and the 
likelihood of BTO status, on the other. Personnel resources are a competing explanation 
for how BTO schools differ from other schools, and these latter findings allow for a 
direct comparison between these alternatives. 

Section 4-A. Description of Beating-the-Odds and Chronically 
Underperforming Schools 

Do BTO and CU schools have different types of staff, students, and perceived climates compared 
to other public middle and high schools? The data suggest that they do. Despite the fact that the 
BTO and CU classification process took student demographics into account, BTO schools had, 
on average, fewer students who received FRPM, and CU schools had more FRPM students than 
other schools. Further, CU schools had a higher proportion of English learners, Hispanic 
students, and African American students compared with BTO and other schools. BTO schools 
were also smaller and CU school larger than other schools. In general, BTO schools had more 
personnel resources than other schools, and CU schools had fewer. Finally, BTO schools had 
markedly more positive school climate than other schools, while CU schools had more negative 
climates. 

Student Demographics of BTO, CU, and Other Public Middle and High 
Schools 

Exhibit 4.A.1 provides demographic characteristics, including enrollment, for BTO, CU, and 
other public middle and high schools in California that administered the CHKS between fall 
2008 and spring 2010. As mentioned in the previous chapter, a school’s student-demographic 
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information was included in the database in the same year that it administered the CHKS. Thus, 
for example, if a school administered the CHKS in spring 2009, its student demographics from 
the 2008-09 school year were represented in the database. 


Exhibit 4.A.I. Demographic characteristics of school groups 


Beating-the-Odds (BTO) Schools 



Middle 
(N = 23) 

Average (%) SD (%) 

High 
(N= 17) 

Average (%) SD (%) 

Total 

(N = 40) 

Average (%) SD (%) 

% FRPM 

43.4 

37.0 

37.9 

32.9 

41.1 

35.0 

% English Learner 

17.2 

17.2 

17.6 

25.6 

17.4 

20.9 

% Hispanic 

45.6 

34.9 

38.6 

33.5 

42.6 

34.1 

% African American 

6.7 

9.7 

4.7 

5.7 

5.9 

8.2 

Enrollment 

750 

240 

860 

606 

796 

431 


Chronically Underperforming (CU) Schools 



Middle 


High 


Total 



(N = 13) 

(N= 7) 


(N = 20) 


Average (%) 

SD (%) 

Average (%) 

SD (%) 

Average (%) 

SD (%) 

% FRPM 

62.5 

20.6 

45.2 

17.1 

56.5 

20.8 

% English Learner 

26.5 

16.6 

18.0 

12.0 

23.6 

15.4 

% Hispanic 

53.3 

23.8 

48.4 

24.8 

51.6 

23.6 

% African American 

10.8 

8.8 

8.1 

3.8 

9.9 

7.4 

Enrollment 

862 

486 

1887 

898 

1220 

809 

Other Public Schools 


Middle 


High 


Total 



(N = 912) 

(N= 732) 

(N = 1 ,644) 


Average (%) 

SD (%) 

Average (%) 

SD (%) 

Average (%) 

SD (%) 

% FRPM 

50.4 

27.0 

41.1 

24.4 

46.3 

26.3 

% English Learner 

19.4 

14.9 

14.8 

11.6 

17.3 

13.7 

% Hispanic 

46.8 

27.6 

42.1 

25.8 

44.7 

26.9 

% African American 

6.7 

8.5 

6.2 

8.2 

6.4 

8.4 

Enrollment 

871 

330 

1395 

653 

1104 

564 


Notes: SD = Standard Deviation 


Overall, the 40 BTO schools were similar to other schools in terms of student race, FRPM 
eligibility, and English learner status, though BTO schools at both levels had slightly lower 
poverty rates, as indicated by FRPM. The largest discrepancy was in average enrollment for 
BTO high schools versus other high schools, with a difference of about 500 students. BTO 
middle schools were smaller, as well, but by a less substantial margin. 


As for the 20 CU schools, they had higher percentages of students in poverty and who were 
English learners, African American, and Hispanic compared to other public middle and high 
schools. CU high schools were also much larger than other schools in terms of enrollment. 6 


6 BTO and CU selection was based on performance residuals, or differences between a school’s actual performance 
and expected performance given its percentage of students were who were FRPM, English learner, Hispanic, 
African American, and recipients of special education services. Thus, schools’ residual performance scores in any 
given year should be uncorrelated with these student demographic variables in that year. However, due to the 
categorical nature of the BTO and CU variables with their cut-points for residual scores and to the multiyear and 
multiple-student-group nature of the BTO criteria, there are still differences in schools’ student demographics 
among BTO, CU, and other schools. Enrollment was not included in the residual performance calculation, perhaps 
explaining its more dramatic fluctuation between school groups. 
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Personnel Resources of BTO, CU, and Other Public Middle and High 
Schools 


In part to confirm the findings of the 2007 SCS report, school group differences in personnel 
resources were explored. As in the 2007 SCS report, personnel resources were separated into (a) 
“resource levels,” which indicate the number of staff per pupil and staff education and 
experience; and (b) “resource ratios,” which indicate the distribution of staff across assignment 
and the status and characteristics of teachers. These variables were drawn from the CBEDS PAIF 
and aggregated to the school level. 7 

Exhibit 4.A.2. Middle and high school personnel resources profile 



Variable/ 




Other 


Resource 


BTO 

CU 

public 


Category 

Variable Description 

schools 

schools 

schools 



Certificated staff per 100 pupils 

6.01 

5.56 

5.78 

C/5 

Staff per pupil 

Teachers per 100 pupils 
Administrators per 100 pupils 

5.19 

0.37 

4.95 

0.33 

5.11 

0.34 

0 

> 


Student support staff per 100 pupils 

0.45 

0.28 

0.34 

_l 

0 


Teachers: average total years of education 

17.5 

17.2 

17.4 

O 

s— 

3 


Teachers: average total years of experience in education 

13.6 

11.6 

13.3 

C/J 

0 

CD 

Education & 

Teachers: average total years of experience in district 

11.2 

9.4 

10.7 

experience 

Administrators: average total years of education 

18.3 

18.1 

18.2 



Administrators: average total years of experience in education 

18.6 

15.3 

16.9 



Administrators: average total years of experience in district 

14.1 

10.7 

11.5 

w 

Distribution of 

Share of total staff who are teachers 

86.5% 

89.0% 

88.5% 

o 

cd 

staff across 

Share of total staff who are administrators 

6.1% 

6.0% 

5.8% 

DC 

0 

assignments 

Share of total staff who are pupil support staff 

7.4% 

5.0% 

5.8% 

o " 

i_ 

3 

o 

Teacher 

Share of total teachers designated probationary or temporary 

17.1% 

23.9% 

16.7% 

C/) 

0 

status & 

Share of total teachers with tenure 

76.2% 

62.8% 

77.7% 

DC 

characteristics 

Share of total teachers with full credentials 

95.9% 

91.8% 

94.9% 


Note: As CBEDS only includes the discrete education attainment level of staff, this study followed the 2007 SCS 


report in creating a continuous education variable in the following way: 

■ Less than bachelor’s degree: 1 2 years of education 

■ Bachelor’s degree: 1 6 years of education 

■ Bachelor’s degree plus 30 or more semester hours: 1 7 years of education 

■ Master’s degree: 1 8 years of education 

■ Master’s degree plus 30 or more semester hours: 19 years of education 

■ Doctorate: 21 years of education 


7 As mentioned in the previous chapter, only those personnel-resources variables that were found to be significantly 
associated with BTO and CU status (p < .05) in the 2007 SCS report were retained for subsequent analyses. 
Administrator experience, although not significantly different in BTO versus CU or other schools in the 2007 report, 
was retained due to the large descriptive difference found in the present database. 
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Exhibit 4.A.2 shows that, on average, CU schools had lower levels of personnel resources than 
BTO or other schools. CU schools had the lowest staff-to-student ratios, the lowest levels of 
education and experience among their staff, and the lowest percentage of teachers who were 
tenured and credentialed. BTO schools had similar resources levels compared to other schools, 
with the exception of more student support staff and more experienced administrators. 

Apart from the student support staff differences between BTO and other schools, these 
descriptive differences paralleled those found in the 2007 SCS report very closely. That report 
found many of these differences to be statistically insignificant. In the present study, they were 
included in the models described below as controls to isolate differences in school climate 
among the school groups. 

School Climate in BTO, CU, and Other Public Middle and High 
Schools 

Exhibit 4.A.3 depicts the differences in school climate indicators across BTO, CU, and other 
public middle and high schools, organized by domain, as described in Chapter 3. 


Exhibit 4.A.3. Middle and high school climate profile 


Variable/ 


BTO 

CU 

Other 

Climate Domain Variable Description 

schools 

schools 

schools 

Overall 

School Climate Index 

346 

247 

299 


Overall supports and engagement 

343 

250 

299 

Supports & 
engagement 

High expectations and caring relationships 

339 

260 

299 

Opportunities for meaningful participation 

336 

263 

299 

Perceived school safety 

346 

246 

299 


School connectedness 

338 

252 

299 


Overall low violence, victimization, and substance use 

344 

254 

299 

Low violence, 

Low physical violence perpetration 

342 

255 

299 

victimization, 
& substance 

Low physical and emotional violence victimization 

341 

266 

299 

use 

Low harassment and bullying 

341 

257 

299 


Low substance use at school 

342 

255 

299 

Low truancy 

Low truancy incidents 

314 

259 

300 


Note: All climate indices range from 100 to 500, with higher scores representing more positive school climates. The 
average score for each indicator is 300, with a standard deviation of 50. 


The SCI and domain and subdomain scores all had means of 300 and standard deviations of 50 
in the statewide sample. Thus, the mean scores for other public schools — which make up over 95 
percent of the sample — were approximately 300 for all climate indicators. Within school groups, 
SCI, domain, and subdomain scores were consistent. Between groups, the differences were stark. 
BTO schools had much higher scores than other schools, almost a full standard deviation across 
all the indicators except low truancy. Expressed in percentiles, the difference in SCI between 
BTO and other schools was 33 percentiles (82 nd and 49 th , respectively). CU schools were 
approximately one standard deviation lower than other schools across all indicators. The 
difference in SCI between BTO and CU schools was two standard deviations, or 68 percentiles 
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(82 nd versus 14 th , respectively), and it was only slightly smaller for other indicators. These are 
dramatic differences by social science standards. 

With magnet schools removed from the sample, the results were almost identical. The largest 
difference was that scores on overall violence, victimization, and substance use were five points 
lower for BTO schools once magnet schools were removed. 

Section 4-B. Regression Analysis of School Climate 
Adjusting for Student Demographics and Personnel 
Resources 

One explanation for this school climate gap among BTO, CU, and other schools might be that 
unmeasured factors underlie both school climate and performance. As shown in Exhibits 4.A.1 
and 4.A.2, BTO schools had smaller enrollments and more teachers per student, fewer students 
in poverty, and more experienced teachers and administrators. These differences may explain 
both BTO status and school climate. In the following analyses, two sets of factors were taken 
into account in an effort to establish the climate-BTO connection more robustly: student 
demographics and personnel resources. 

Exhibit 4.B.I. Regression results for school climate outcomes ( N = 1 ,657) 


Variable/ 

Climate 


BTO 

CU 


Category 

Variable Description 

p 

SE 

p 

SE 

R 2 

Overall 

School Climate Index 

37.8*** 

7.6 

-33.6*** 

8.0 

0.44 


Overall supports and engagement 

37.3*** 

7.5 

-33.9*** 

7.2 

0.42 

Supports & 
engagement 

High expectations and caring relationships 

30.8*** 

8.1 

-26.0** 

9.7 

0.28 

Opportunities for meaningful participation 

26.9** 

8.9 

-20.8* 

8.1 

0.38 

Perceived school safety 

36.2*** 

7.1 

-33.5*** 

6.9 

0.48 


School connectedness 

29.6*** 

7.0 

-29.0*** 

7.3 

0.39 


Overall low violence, victimization, and substance use 

38.2*** 

8.5 

-28.9*** 

8.0 

0.34 

Low violence, 

Low physical violence perpetration 

36.7*** 

8.5 

-26.7*** 

7.6 

0.38 

victimization, & 

Low physical and emotional violence victimization 

37.6*** 

9.1 

-24.9** 

8.8 

0.16 

substance use 

Low harassment and bullying 

37.0*** 

8.0 

-27.0** 

9.4 

0.24 


Low substance use at school 

37.0*** 

8.1 

-25.9** 

8.6 

0.40 

Low truancy 

Low truancy incidents 

12.3* 

6.2 

-31.2* 

14.8 

0.18 


*= pc. 05 **= pc. 01 *** = pc. 001 

Notes: All models included the following variables as controls: the proportion of students who are FRPM, African 
American, Hispanic, and English learners; enrollment; average teacher education and experience; the proportion of 
tenured and fully credentialed teachers; average administrator experience; and the proportion of total staff who are in 
student support services roles. p is the regression coefficient estimating the relationship of BTO/CU status and the 
indicated outcome, and SE is the standard error. R 2 is the proportion of overall variance in the outcome explained by 
all model variables. All models were estimated using robust standard errors to adjust for heteroskedasticity. 

The results of the multivariate regression models of school climate on BTO and CU status, 
controlling for student demographics and personnel resources, are shown in Exhibit 4.B.I. The 
results largely confirmed the differences in school climate across BTO, CU, and other schools 
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that are shown in Exhibit 4.A.3. Across all school climate indicators, there was a statistically 
significant difference among BTO, CU, and other schools. In other words, student demographics 
and personnel resources being equal, BTO schools still had more positive climates and CU 
school less positive climates than other schools. 

BTO schools were predicted to have an SCI 37.8 points higher than a regular school and 71.4 
higher than a CU school, after taking into account student demographics and personnel 
resources. Considering that the standard deviation of the SCI is 50, this implies that BTO schools 
had levels of school climate that were 1.4 standard deviations above CU schools and 0.8 standard 
deviations above other schools with identical enrollments, student demographics, and staff 
resources. These are large differences by social science standards. For each of the two school 
climate domains — supports and engagement; and low violence, victimization, and substance 
use — the estimated differences by school group were almost identical. 

The subdomains, too, were estimated to have roughly similar differences among BTO, CU, and 
other schools. BTO schools had the largest separation from other schools in terms of the low- 
physical-and-emotional-violence-victimization subdomain (/? = 37.6, p < .001). The smallest 
separation was for opportunities for meaningful participation (/? = 26.9, p < .01), but even this 
difference is considered medium to large (Cohen, 1988). The largest and smallest differences for 
CU schools versus other schools were in the perceived-school-safety (J3 = -33.5, p < .01) and 
opportunities-for-meaningful-participation (/? = -20.8, p < .05) subdomains, respectively. 

With magnet schools removed from the sample, the substantive interpretation of the results was 
the same. The coefficients for BTO classification were attenuated in each of the models, but they 
were all still significant at the p <.01 level and all still larger than /? = .20. 

School Climate Differences by API Level 

Do these differences in climate hold, however, for BTO and CU schools at different levels of 
academic performance? For example, looking only at schools with lower APIs, do BTO and CU 
schools have different school climates than other schools? 

There were too few BTO and CU schools to estimate separate regression models by API 
quartiles, but Exhibit 4.B.2 gives a descriptive glance at school-group climate differences by 
2008 or 2009 API (as with other CBEDS data, schools’ APIs from the school year corresponding 
with their CHKS administration were used). The first quartile of API, for example, represents 
schools with the lowest academic performance; the fourth quartile represents schools with the 
highest. The exhibit shows that the differences in school climate between BTO and other schools 
were upheld in schools in the first, second, and fourth quartiles of API. That is to say, BTO 
schools had more positive school climates than other schools among both high- and low- 
performing schools. 8 

There were no CU schools in the fourth API quartile, as one might suspect, given that 
chronically low-performing schools are unlikely to turn out high-performance scores in any 


8 For schools in the third API quartile, the BTO schools actually had lower average SCIs than other schools. 
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given year. However, CU schools had more negative climates than other schools in the bottom 
three quartiles. 


Exhibit 4.B.2. Box plot of average SCI, by school group and 2008-2009 API quartile 
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BTO CU Other BTO CU Other BTO CU Other BTO CU Other 


Note: 1 = 1 st quartile of school based on API (i.e., lowest performing quartile); 2 = 2 nd quartile; 3 = 3 rd quartile; 4 = 
4 th quartile (i.e., highest performing quartile). 


The results of the regression analyses and the description of school-group climate differences 
stratified by API have several implications. First, there were significant medium-to-large 
differences in school climate among BTO, CU, and other schools. BTO schools had more 
positive climates than other schools, and CU schools had more negative climates. School climate 
helps to answer the question, “What is different about BTO and CU schools?” A nuanced look at 
these relationships in schools at different API levels suggested that school climate may help 
differentiate BTO, CU, and other schools across levels of academic performance. This latter 
finding should be interpreted with caution, however, as there were very few BTO schools in 
certain API quartiles (and no CU schools in the highest quartile), rendering the depicted climate 
differences in some quartiles more susceptible to random error. 


Section 4-C. Relative Associations of School Characteristics 
and Likelihood of Beating the Odds 

The previous set of regression analyses estimated the differences in school climate by school 
group, controlling for student demographics and personnel resources. This section presents the 
results of a logistic regression analysis that took a slightly different look at the associations 
between each of these factors. Here, school climate, student demographics, and personnel 
resources were each treated as predictors of a school’s likelihood of beating the odds. 

This analysis, the results of which are shown in Exhibit 4.C.1, juxtaposed the relative 
associations of climate, demographics, and resources with the likelihood of beating the odds. The 
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beta coefficients presented in the exhibit (ft) are in “log-odds” units, which have little meaning in 
real-world terms. However, they do allow for a direct comparison of the strength of association 
between the model predictor variables and the likelihood of beating the odds. The results 
suggested that a school’s SCI had the strongest association with the likelihood of beating the 
odds (J3 = 1.19, p < .001). A school’s personnel resources (/? = 0.49, p < .01) and student 
demographics = 1.10, p < .001) also had significant associations with the likelihood of beating 
the odds. It should be noted that the association between student demographics and BTO was 
driven almost entirely by enrollment, which was negatively related to the likelihood of BTO 
classification. 


Exhibit 4.C.1 shows that the probability 9 of being classified as BTO was only 1.1 percent in an 
average school (i.e., when all predictor variables were at their sample mean value). In other 
words, the probability of beating the odds was very low. However, the results suggested that, in 
an otherwise average school, SCI was associated with the following probabilities of BTO: 

• An SCI one standard deviation above the mean was associated with a 3.5 percent 
probability of beating the odds. 

• Two standard deviations above the mean equated to a 10.6 percent probability of beating 
the odds. 

• A school with an SCI equal to the highest SCI value in the sample had a 44.5 percent 
probability of beating the odds. 

While the likelihood of beating the odds was low for all public middle and high schools in 
California, an exceptional school climate was associated with an exponentially higher 
probability. The results suggested that a school with a climate two standard deviations higher 
than the average school was more than ten times likely to be BTO. This probability was even 
greater for schools with the best climates. With magnet schools removed from the sample, the 
results were almost identical. 

Exhibit 4.C.I. Logistic regression results for BTO outcome ( N = 1,704) 



tt 

SE 

BTO 

probability 
at mean* 

BTO 

probability 
at +1SD* 

BTO 

probability 
at +2SD* 

BTO 

probability 
at max* 

School Climate lndex § 

1 19 *** 

0.19 

1.1% when all 

3.5% 

10.6% 

44.5% 

Personnel resources 

0.49 ** 

0.14 

predictors are 

1 .7% 

2.8% 

3.9% 

Student demographics 

1.10 *** 

0.18 

at mean value 

3.2% 

9.0% 

32.4% 


* p<. 05 ** p<.0l *** p<.001 

t Coefficients are in log-odds units. 

j These values reflect the probability of a school being classified as a BTO school, given the indicated value of the 
predictor variable (e.g., plus one standard-deviation [SD]). These probabilities assume all other predictor variables 
are at their sample mean value. 

§ All predictor variables are standardized (i.e., mean = 0, standard deviation = 1). 

Notes: /3 is the regression coefficient estimating the relationship of the indicated predictor and BTO likelihood, and SE 
is the standard error. +1 SD and +2SD refer to “one standard deviation above the mean” and “two standard deviations 
above the mean,” respectively. 


9 Because coefficients presented in log odds are difficult to interpret in meaningful terms, probabilities can be 
estimated from the results of logistic regression models that indicate the probability of the outcome being true (i.e., 
of a school being a BTO school), given specified values of the predictor variables. 
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The results suggested that school climate had more to do with BTO than personnel resources. 
The effect size for school climate was more than twice as large. Even in a school with the 
optimal level of personnel resources, the probability of beating the odds was still only 3.9 
percent. For student demographics the association with BTO probability was not significantly 
different from that of school climate. 10 Enrollment appeared to be the main driver of the 
association between student demographics and BTO, with smaller student bodies predicting a 
greater likelihood of BTO. 


10 Based on a post-estimation Wald test (j = 0.15, n.s.). 
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Chapter 5. Conclusions 


This study adds to the growing body of evidence that suggests that school climate is one of the 
factors that differentiates schools that succeed from those that do not. There were significant 
differences in school climate between successful and unsuccessful schools, even when taking 
into account schools’ student characteristics and resources. Interpreted in successful-schools- 
approach terms, it could be said that the attributes of climate evidenced in these schools that are 
BTO represent a model for all schools. Helping foster similarly positive school climate should be 
part of efforts to improve the performance of public schools. 

Furthermore, the study indicates that a school’s climate may have more to do with its success 
than the resources at its disposal. 11 This result implies that things like high expectations for 
students, caring relationships between teachers and students, and feeling safe at school are more 
associated with success than teacher or administrator experience or student support services staff 
ratios. In two schools with identical staff in terms of roles, experiences, and education, the 
school with a better climate has a much greater probability of being successful, according to this 
study’s findings. 

These two sets of factors — school climate and personnel resources — are potentially related. The 
literature reviewed in Chapter 2 pointed to both resources (e.g., teacher experience) and climate 
(e.g., high expectations) as elements of effective schools. There may be a mediating relationship 
whereby more experienced and educated teachers and administrators have more capacity for 
creating an environment characterized by connectedness, caring, and safety, which in turn fosters 
success. The interconnections between these factors should be the subject of future research. 

Lastly, enrollment appears to have an important role in school success. The results of this study 
imply that BTO schools have significantly smaller enrollments and that CU schools have 
significantly larger enrollments. As with personnel resources, future research could examine the 
interconnections (e.g., mediating or moderating relationships) between enrollment and school 
climate. What the present results do suggest, however, is that in two schools of the same size, 
school climate is still significantly associated with the likelihood of BTO. 

The authors have been careful throughout this report in referring to the relationship between 
school climate and success as an “association” rather than an “effect.” It is tempting to assert 
causality in the relationship between these variables. However, the present analyses are cross- 
sectional and not experimental, and while they address some alternative explanations for why 
school climate and performance are related, many potential confounds are unaccounted for. It 
could be that a third, unmeasured variable explains both school climate and success. For 


11 In the successful-schools approach, if the findings of a successful-schools analysis yield insignificant differences 
in resource use between successful and unsuccessful schools, the implication is that all schools should be able to 
succeed with the resources that they have (see, for example, Augenblick & Myers, 2001). That is, all schools’ 
spending levels are adequate for success. According to this logic, if successful schools are spending no more than 
unsuccessful schools, then spending alone is not the answer. Other factors must differentiate schools that succeed 
from those that do not. This study indicates school climate is one of these factors. 
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example, in schools where students’ parents are highly involved, parent engagement may be the 
true engine behind academic success and a positive school climate. 

The directionality of the relationship between climate and success is also in question. When a 
school consistently beats the odds in terms of academic performance, it may engender a positive 
school climate. Nonetheless, it is clear from the results of this study that BTO schools have 
significantly more positive school climates, adding to the large and growing body of research 
demonstrating that safe and supported students perform better in school. 

Implications for Practice 

Students’ demographics are strong predictors of their academic success. While some educational 
initiatives have begun targeting the social forces that make poor, racial minority, and English 
learners less successful in school (see Ladd, 2012), educators can take more immediate measures 
to meet all students where they are and improve their learning. 

The factors explored in this study — school climate, staffing, and school size — are all malleable 
factors, things that schools or districts can manipulate. Student characteristics — things like race, 
socioeconomic status, first language, and special needs — and the factors that put students of 
certain demographics at a disadvantage are infeasible for schools and districts alone to 
manipulate. The approach employed in this study helps direct the focus toward characteristics of 
schools that can be targeted and changed through intervention. School climate is a particularly 
promising area of focus. 

Among the more commonly employed strategies for improving school climate are schoolwide 
prevention approaches that involve all staff in the installation of a behavior management system 
and student social and emotional learning approaches that rely on classroom social-skills 
instruction (Osher, Bear, Sprague, & Doyle, 2010). For example, some schools have 
implemented restorative justice programs as alternatives to traditional, punitive discipline codes. 
These programs focus on the relationship between the perpetrator of misbehavior and members 
of the school community, including potential victims and their families. Another example is 
transformative classroom management, which leverages students’ motivation and engagement to 
increase adherence to classroom behavioral norms. Expansion of these types of programs and 
increased support through mechanisms like the U.S. Department of Education’s Safe and 
Supportive Schools program could help improve climate in more schools. 

This study suggests that a positive school climate is an asset for all schools, serving all types of 
students across demographic spectrums. This news is especially encouraging for schools whose 
students face socioeconomic barriers to academic success. These schools must beat the odds to 
achieve conventional levels of success. Fortunately, as this study shows, certain schools are 
beating the odds, and these “pockets of excellence” have a positive school climate in common. 
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Appendix A. Beating-the-Odds Schools 


The names of the 40 California public middle and high schools that were identified in this study 
as “beating the odds” are shown below. 


School 

District 

% FRPM* 

1. 

Abraham Lincoln Middle 

Selma Unified 

74 

2. 

Anderson W. Clark Magnet High 

Glendale Unified 

44 

3. 

Arizona Middle 

Alvord Unified 

66 

4. 

Avenal High 

Reef-Sunset Unified 

99 

5. 

Bonita Vista Senior High 

Sweetwater Union High 

19 

6. 

Bud Carson Middle 

Hawthorne 

76 

7. 

C. T. English Middle 

Loma Prieta Joint Union Elementary 

4 

8. 

California Academy of Mathematics and Science 

Long Beach Unified 

37 

9. 

Crescenta Valley High 

Glendale Unified 

9 

10. 

Day Creek Intermediate 

Etiwanda Elementary 

10 

11. 

Dr. T. J. Owens Gilroy Early College Academy 

Gilroy Unified 

30 

12. 

Edison Computech 

Fresno Unified 

59 

13. 

Foothill Technology High 

Ventura Unified 

18 

14. 

General Grant Middle 

Kings Canyon Joint Unified 

100 

15. 

Georgina P. Blach Junior High 

Los Altos Elementary 

1 

16. 

Granite Ridge Intermediate 

Clovis Unified 

16 

17. 

Hawthorne Middle 

Hawthorne 

74 

18. 

Huntington Middle 

San Marino Unified 

1 

19. 

Joaquin Moraga Intermediate 

Moraga Elementary 

1 

20. 

Kearny International Business 

San Diego Unified 

75 

21. 

Mendota High 

Mendota Unified 

95 

22. 

National City Middle 

Sweetwater Union High 

79 

23. 

Newton Middle 

Hacienda la Puente Unified 

51 

24. 

Oak Park High 

Oak Park Unified 

2 

25. 

Orinda Intermediate 

Orinda Union Elementary 

0 

26. 

Palos Verdes High 

Palos Verdes Peninsula Unified 

2 

27. 

Palos Verdes Intermediate 

Palos Verdes Peninsula Unified 

1 

28. 

Piedmont High 

Piedmont City Unified 

0 

29. 

Piedmont Middle 

Piedmont City Unified 

0 

30. 

Prairie Vista Middle 

Hawthorne 

76 

31. 

Richardson Prep Hi 

San Bernardino City Unified 

70 

32. 

Roosevelt International Middle 

San Diego Unified 

77 

33. 

San Diego International Studies 

San Diego Unified 

46 

34. 

San Diego Science and Technology 

San Diego Unified 

74 

35. 

Saratoga High 

Los Gatos-Saratoga Joint Unified 

0 

36. 

Segerstrom High 

Santa Ana Unified 

58 

37. 

Sierra Vista Middle 

Hacienda la Puente Unified 

82 

38. 

Sparks Middle 

Hacienda la Puente Unified 

80 

39. 

Stone Valley Middle 

San Ramon Valley Unified 

1 

40. 

West Campus 

Sacramento City Unified 

37 


* Percentage of students eligible for the free and reduced-price meals program 


The sample for this study was limited to schools that administered the California Healthy Kids 
Survey during the 2008-09 and 2009-10 school years. During this period, the survey was 
administered in a representative sample of 1,715 public middle and high schools, roughly two 
thirds of the state total. Schools that did not administer the survey during this period were not 
eligible to be labeled as “beating the odds” in this study. Eight of the 40 (20 percent) are 
classified as magnet schools by the National Center for Education Statistics, whereas only five 
percent of other sample schools are magnet schools. Parallel analyses were conducted with 
magnet schools excluded, and there were no substantive differences in results. 
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